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(54) Free form ink reminders 

(57) A method and apparatus for generating remind- 
er messages based on a freeform input is disclosed. In 
one embodiment, freeform inputs are handwritten anno- 
tations on paper (10) that are scanned into a computer 
system (100). Based on information contained in the 
freeform input, the system creates a reminder message 



and transmits it in response to a time reference con- 
tained in the freeform input, in an alternative embodi- 
ment, freeform digital ink inputs may be directly entered 
into a pen computer system via a stylus. Reminder mes- 
sages may be delivered over a computer display, fac- 
simile device, via email, telephone or portable paging 
device. 
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Description 

[0001] The present invention relates generally to a 
schedule organizing system. In particular, the present 
invention relates to a method and apparatus for gener- s 
ating messages based on freeform input supplied by a 
user. 

[0002] U.S. patent application 09/094,730, A Method 
and System for Creating Ad Hoc Links From Freeform 
Ink, discloses a paper-like hypertext system that sup- 10 
ports reading and browsing and creates ad hocWnks be- 
tween passages in documents using user-specified 
freeform ink annotations. However it does not disclose 
a system for generating reminder messages based on 
freeform inputs. is 
[0003] U.S. patent application 08/929,427, A Method 
and System for Organizing Documents Based on Anno- 
tations in Context, discloses a method and system for 
organizing documents based on freeform annotations 
made to the documents. However, It does not disclose 20 
a system for generating reminder messages based on 
freeform inputs. 

[0004] People often require the use of organizational 
tools to help manage schedules and to keep track of ap- 
pointments and other events. A paper calendar is a com- 25 
mon example of such an organizational tool. A person 
may write on the calendar the~date arid time of each en- 
try. The person can also denote reminders related to the 
entry, thus providing sufficient lead time to prepare for 
a scheduled event. Day timers are another example of 30 
a paper-based organizational tool. Day timers keep 
track of names, appointments, and lists of to do items, 
' and often employ a calendar system. As with the com- 
mon paper calendar, day timers are very easy to use: 
the person makes notes on the appropriate page or cal- 35 
endar entry Lead times are also easily provided by plac- 
ing reminders on preceding pages. Both systems share 
the advantages of a large, dedicated space to show 
notes and annotations, portability, and the use of hand- 
written annotations. However, one disadvantage of pa- <o 
per-based organizational tools is that they do not pro- 
vide an automatic reminder mechanism. The time to go 
to a meeting can easily slip by though they entered the 
event in a calendar or daytime and included a lead time 
reminder. 4$ 
[0005] Electronic calendars provide a different ap- 
proach to schedule organization. Electronic calendars 
run on desktop, laptop, and other computers. The main 
advantage of electronic calendars is that they provide 
an automatic reminder mechanism. They may program so 
the computer to display a calendar entry such as an ap- 
pointment or reminder entry at a designated time. With- 
out any additional action by the user, the reminder mes- 
sage automatically appears on the computer screen at 
a designated time. The main disadvantage is that oper- ss 
ating an electronic calendar organizing system requires 
a lot of effort: select tim , enter event description text, 
select options, etc. There is no facility for a user to enter 
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freeform input, such as a handwritten annotation. Simi- 
larly, an electronic calendar system do s not work when 
a laptop computer is turned off. 
[0006] There is a need for a schedule organizing sys- 
tem that provides the simplicity of freeform annotation 
of paper-based systems with the automatic reminder 
mechanism of electronic calendars. 
[0007] In an embodiment of the present invention a 
method and apparatus for generating reminder messag- 
es based on a freeform input received from a user is 
provided. Annotations can be made by hand to a paper 
calendar and scanned mto a computer system. Based 
on the freeform annotation, the system automaticalfy 
schedules and issues the corresponding reminder mes- 
sages without any additional user interaction. In an al- 
ternative embodiment, freeform digital ink annotations 
may be directly entered into the system via a pen com- 
puter interface. 

[0008] In one aspect, an embodiment of the present 
invention provides a method lor providing an alert based 
on freeform input, comprising the steps of accepting a 
freeform input where the input contains message infor- 
mation and a time reference when a reminder message 
is to be provided; extracting the message information 
and time reference from the freeform input; and provid- 
ing Jhe alert responsive to the time reference. 
[0009] In another aspect, an embodiment of the 
present invention provides a system for providing an 
alert based on a freeform input, the system comprising 
a receiving device for receiving a freeform input, the 
freeform input containing message information and a 
time reference for delivery of said message; a processor 
coupled to the receiving device for extracting the mes- 
sage information and time reference from the freeform 
input; and an alerting device for providing an alert to the 
user in response to said time reference. 
[0010] In another aspect, an embodiment of the 
present Invention provides a computer readable storage 
medium for use with a computer apparatus, wherein the 
medium includes computer instructions which, when ex- 
ecuted by said computer apparatus: receive a freeform 
input which includes message information and a time 
reference; extract information from the freeform input; 
and provide an alert responsive to the time reference. 
[0011] The present invention will be described with re- 
spect to particular embodiments thereof, and reference 
will be made to the drawings in which: 

Fig. 1 illustrates the major components of an em- 
bodiment of the present invention and associated 
devices; 

Figs. 2A and 2B illustrate an embodiment of the 
present invention implemented on a pen-based 
computer; 

Fig. 3 is an illustration of a page from a paper cal- 
endar thai may be used as an input with the present 
invention; 

Fig. 4 is a flowchart illustrating the logical sequence 
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of steps performed to achiev stroke grouping in an 
embodiment of the present invention; 
Fig. 5 is a flow chart illustrating the logical sequence 
of steps required to perform the field finding function 
of the present invention; 

Fig. 6 is a flow chart illustrating the logical sequence 
of steps required to perform the reminder message 
function of the present invention; and 
Fig. 7 is a block diagram illustrating the major com- 
ponents of a computer system utilized by the 
present invention. 

[0012] Directing attention to Fig. 1, in one preferred 
embodiment, the present invention allows the user to 
schedule appointments on a paper calendar 10 and in- 
put them to a computer system 100 via a scanner or 
facsimile device 30. Scanner device 30 communicates 
with the computer system 100 via communication me- 
dium 35. The computer system 1 00 is a general purpose 
computer that generates an alert in the form of a remind- 
er message. The reminder message is based on the an- 
notation 22 on calendar page 10 and is sent at a time 
indicated on calendar page 10. A typical general pur- 
pose computer will be discussed below. As used herein, 
the term •freeform input" refers to idiosyncratic marks, 
such as'hand-writteri annotations. When displaying the 
message on the display 185, the freeform input 22 is 
reproduced on the display 185. The reminder message 
mday also be presented to the user via telephony de- 
vices such as facsimile device 30, or portable pager 60 
via communication medium 55. 
[0013] Besides displaying the freeform input 22, an 
embodiment of the present invention may also utilize a 
handwriting recognition algorithm which recognizes the 
handwritten characters and forms a corresponding au- 
ditory reminder message. Handwriting recognition algo- 
rithms are widely known by those skilled in the art and 
need not be disclosed herein. When an auditory remind- 
er message is created, the computer system 100 may 
send the reminder message via communication medium 
45 to telephone 50, or by an audio system 150 incorpo- 
rated in computer system 1 00. 
[001 4] I n an alternative preferred embodiment (Figs. 
2A and 2B), the present invention may also utilize a 
touch-screen or pen-based computer system 101. Pen- 
based computer systems present a familiar paper-like 
interface loan electronic calendar In a pen-based com- 
puter system such as system 101 . the user writes with 
a stylus 187 directly on display 186. The computer sys- 
tem 1 0 1 senses the locat ton and the positions traversed 
as the stylus 1 87 moves across the display 1 86 and as- 
signs 'digital ink' marks that correspond with the posi- 
tions of the stylus 187. As used herein, the term "digital 
ink* refers to traces that appear on the computer display 
186, namely collections of pixels that are darkened or 
colored to represent the strokes that the user made with 
a stylus. In this alternative preferred embodiment, a 
page similar to the paper calendar is reproduced on the 



display 186 and the user may make freeform annota- 
tions directly to it via the stylus 187. 
[0015] Fig. 3 illustrates a typical calendar form 10 con- 
taining freeform annotations. Date 14 may also be in- 
5 eluded at the top of the page. In column 16, various 
times of the day are listed, allowing a user to organize 
appointments or other events throughout the day. Col- 
umn 18 provides a space for the user to place a dash 
or callout symbol that flags that time of day for a remind- 

io er message. Adjacent to column 18 is column 20, which 
allows space for reminder notes. An example is input 
22, entered in the space corresponding to 12 p.m. on 
Thursday, February 2nd. In this illustration, the freeform 
input 22, 'lunch with Chelsea 1 is placed to the right of a 

'5 dash in column 1 8, which causes at 1 2 p.m. on February 
2nd a reminder message containing the words 'lunch 
with Chelsea" to be displayed on the computer display 
185 or output to other devices discussed above. 
[0018] Where the reminder message is outputtocom- 

?0 puter display 185 or the display 186 on a pen-based 
computer system 101 , the user may also associate oth- 
er documents with the reminder message. For instance, 
in the example of the "lunch with Chelsea" reminder 
message, a user may wish to remind himself of a list of 

25 notes to be discussed during lunch. If the user had pre- 
pared the list of notes and either scanned or faxed them 
into the system or prepared the document on a word 
processor, the user may associate that document with 
the 1 2 p.m. timestot by placing an icon 24 on the image 

30 of the calendar page 1 0 stored in the computer system 
100 (or computer system 101). When the reminder mes- 
sage is displayed at the appropriate time on the compu- 
ter display 185, the document or a link to it can be dis- 
played as well. 

& [001 7] The present invention executes program 300X 
to issue a reminder message based on a freeform input. 
Directing attention to Fig. 4, at step 310, the program 
300X determines whether the freeform input is a 
scanned image or strokes on a pen computer 187. If it 

40 is a scanned image, control proceeds to step 31 2, where 
annotation lifting is performed. Annotation lifting refers 
to the extraction of the freeform input from the scanned 
image. Annotation lifting converts the scanned image, 
which contains both the printed calendar page 10 and 

4$ the freeform input 22, into an image containing only the 
freeform input. The logic of the annotation lifting imple- 
mented here is similar to the extraction algorithm de- 
scribed in U.S. Patent 5,692,073, Formless Forms and 
Paper Web Using a Reference-Based Mark Extraction 

so Technique, which is herein incorporated by reference in 
its entirety. Once the annotation lifting step 312 has 
been performed, program control continues to step 314, 
where thinning is performed on the extracted freeform 
input. Thinning is a process wherein an image is con- 

& verted intoa listof lines or curve segments, and features 
in the image that do not form lines or curve segments 
are ignored as noise. Each curve segment is equivalent 
to a stroke, except that the direction assigned is arbitrary 



3 



5 

and there is no timing information. Furthermore, the or- 
dering of the curve segments is arbitrary, whereas the 
order of strokes in a pen -based computer 101 is implicit 
in the user's actions. Thinning is known to those skilled 
in the art and need not be described herein. 
[0018] Directing attention back to step 310, if the input 
mode is strokes made on a pen computer 187 with a 
stylus 101, control proceeds to step 316, where the 
strokes are captured to the pen computer 1B7. 
[0019] Once the freeform input has been obtained 
through either of the methods described above, control 
proceeds to 6tep 318, where the individual strokes are 
grouped together. As used herein, a stroke refers to the 
stroke of a stylus or a curve segment produced by thin- 
ning, and may be represented in a computer as a series 
of connected dots. Stroke grouping is known to those 
skilled in the art and need not be discussed herein. Once 
stroke grouping is performed, control proceeds to step 
320, where field finding is performed. As used herein, a 
"field" is a logical structure that corresponds to a two 
dimensional region on the form. The field finding proc- 
ess detects the appropriate field that is associated with 
the stroke groups. Step 320 is described in greater detail 
in Fig. 5. 

[0020] After the field finding step 320 is performed, 
control proceeds to step 322, where the message is 
generated and delivered. 

[0021] Fig. 5 shows the logical sequence of steps ex- 
ecuted to find fields corresponding to the freeform input. 
At step 500, the first stroke group is obtained. At step 
501 , a bounding rectangle is defined to be the area sur- 
rounding the stroke group. At step 502, the target field 
is initialized. A target field is a variable that represents 
afield that received some strokes belonging to the free- 
form input. Continuing to step 504, a target overlap is 
initialized. A target overlap ie a variable that represents 
the amount of overlap between the bounding area con- 
taining the stroke and the area of the field. Control con- 
tinues to step 505, where the current field is set to the 
first field. Continuing to step 506, an additional variable, 
intersection extent, is set to the extent of the intersection 
between the bounding rectangle and the area of the cur- 
rent field. At step 508, the intersection extent is com- 
pared to the target overlap extent. If the intersection ex- 
tent is greater than the target overlap extent, control pro- 
ceeds to step 510 where the target overlap extent is set 
to the value of the intersection extent, and the target field 
is set to the current field (step 512). Control then pro- 
ceeds to step 514, where a check is made for any re- 
maining fields. II there are remaining fields, the current 
field is set to the next field (step 51 5), and control returns 
to step 506. Otherwise, control continues to step 516, 
where the event time is given the value corresponding 
to the target field. The time value is taken from the field 
in column 16 corresponding to the target field in column 
20 (Fig.2). If multiple lines are needed for the messag 
in column 20, a mark may be placed in column 18 cor- 
responding to the desired time reference in column 16. 
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Control continues to step 518, where the lead time for 
the reminder message is obtained. The lead time is the 
time that the user indicates a first alert is to be issued, 
before the event time, thus allowing the user a sufficient 
5 amount of time to prepare for the upcoming event. The 
lead time may be designated by the user by marking in 
column 1 8 (Fig. 4). The vertical extent and the shape of 
the mark may be used to determine the lead time. Con- 
tinuing to step 520, the freeform input 22 is checked for 
w any additional stroke groups. If any are found, control 
proceeds to step 524, where the next stroke group is 
selected. Control returns to step 501 . 
[0022] Messages are generated as described in Fig- 
ure 6. Beginning at step 600 the clock 165 is polled pe- 
is riodically until it is time to produce a reminder message. 
At the time the user is to be notified, the method of de- 
livery is determined. Where the notification mechanism 
is the display device of a computer system (step 602) 
such as display 185, display 186, or any other graphics 
20 display, control proceeds to step 604, where the remind- 
er message is shown on the display device. This may 
also include sending the reminder message through a 
network to another computer, via e-mail or other suitable 
communications medium. Control then returns to step 
25 600. If the notification mechanism is facsimile device 30 
(step 606) control proceeds to step 608, where the re- 
minder message clipped from the calendar page 10 
along with the event time is transmitted to facsimile de- 
vice 30. Control then returns to step 600. With reference 
30 to step 606, If the delivery mechanism is not facsimile, 
control proceeds to step 610, where freeform input 22 
corresponding to the reminder message is converted to 
equivalent ASCII characters using handwriting recogni- 
tion algorithms that are known in the art. If the conver- 
ts sion is successful (step 612), control proceeds to step 
616. If the conversion is not successful (6tep 612), con- 
trol proceeds to step 61 4 where a generic reminder mes- 
sage is generated based on the time of event, and con- 
trol transitions to step 616. If the delivery mechanism is 
40 voice-based, such as telephone 50, control proceeds to 
step 61 8, where voice is generated from the ASCI I char- 
acters. Control then proceeds to step 620 where the 
voice message is sent. Control then returns to step 600. 
Directing attention back to step 616, if the delivery 
45 mechanism is not voice, control proceeds to step 622. 
If delivery mechanism is a telephony device such as a 
telephone 50 or pager 60 that is capable of communi- 
cating visual text (step 606), control proceeds to step 
624 where the message is delivered to the telephony 
50 device. Control then returns to step 600. 

[0023] Fig. 7 is high-level block diagram view of an 
embodiment of a computer system having a computer 
program that causes the computer system to perform 
the method of the present invention. The computer sys- 
55 tern 146 includes a processor 130 and memory 125. 
Processor 130 may contain a single microprocessor, or 
may contain a plurality of microprocessors lor configur- 
ing the computer system as a multi-processor system. 
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Memory 125, stores, in part, instructions and data for 
execution by processor 1 30. If the system of the present 
invention is wholly or partially implemented in software, 
including a computer program 300X, memory 125 
stores the executable code when in operation. Memory 
125 may include banks of dynamic random access 
memory (DRAM) as well as high speed cache memory. 
The system 1 00 further includes a mass storage device 
1 35, peripheral device(s) 1 40, input device(s) 1 55, port- 
able storage medium drive(s) 160, a graphics subsys- 
tem 170 and a display 185. In Figure 1 B, display 166 is 
substituted for display 185 and stylus 187 is substituted 
for input device 155. For simplicity, the components 
shown in Fig. 7 are depicted as being connected via a 
single bus 180. However, the components may be con- 
nected through one or more data transport means. For 
example, processor 130 and memory 125 may be con- 
nected via a local microprocessor bus, and the mass 
storage device 135, peripheral device(s) 140, portable 
storage medium drrve(s) 160, and graphics subsystem 
170 may be connected via one or more input/output (1/ 
O) buses. Mass storage device 135, which is typically 
implemented with a magnetic disk drive or an optical 
disk drive, is a non-volatile storage device for storing 
data and instructions for use by processor 130. The 
method of the present invention also may_be stored in 
processor 130. Portable storage medium drive 160 op- 
erates in conjunction with a portable non -volatile stor- 
age medium, such as a floppy disk, or other computer- 
readable medium, to input and output data and code to 
and from the computer system 1 00. In one embodiment, 
the method of the present invention is stored on such a 
portable medium, and is input to the computer system 
100 via the portable storage medium drive 160. Periph- 
eral device(s) 140 may include any type of computer 
support device, such as an input/output (I/O) interface, 
to add additional functionality to the computer system 
100. For example, peripheral device(s) 140 may include 
a network interface card for interfacing computer system 
100 to a network, a modem, and the like. Input device 
(s) 1 55 provide a portion of a user interface. Input device 
(s) 155 may include an alpha-numeric keypad for input- 
ting alpha-numeric and other key information, or a point- 
ing device, such as a mouse, a trackball, stylus or cursor 
direction keys. In order to display textual and graphical 
information, the computer system 1 00 includes graphics 
subsystem 170 and display 185. Display 185 may in- 
clude a cathode ray tube (CRT) display, liquid crystal 
display (LCD), other suitable display devices, or means 
for displaying, that enables a user to interact with the 
computer program. Graphics subsystem 170 receives 
textual and graphical information and processes the in- 
formation for output to display 185. Additionally, the sys- 
tem 100 may include output devices 145. Examples of 
suitable output devices include speakers, printers, and 
the like. The devices contained in th computer system 
100 are those typically found in general purpose com- 
puter systems, and are intended to represent a broad 
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category of such computer components that are well 
known in the art. The computer syst m of Fig. 7 illus- 
trates one platform which can be used for practically im- 
plementing the method of the present invention. Numer- 

5 ous other platforms can also suffice, such as Macintosh- 
based platforms available from Apple Computer, Inc., 
platforms with different bus configurations, networked 
platforms, rnulti- processor platforms, other personal 
computers, workstations, mainframes, and the like. Al- 

10 ternative embodiments of the use of the method of the 
present invention in conjunction with the computer sys- 
tem 1 00 further include using other display means, such 
as CRT display, LCD display, projection displays, or the 
like. Likewise, any similar type of memory, other than 

1$ memory 125, may be used. Other interface apparatus, 
in addition to the component interfaces, may also be 
used including alpha-numeric keypads, other key infor- 
mation or any pointing devices such as a mouse, track- 
ball, stylus, cursor or direction key. 

20 [0024] Messages may be delivered by email. 



Claims 

» 1. A method for providing an alert based on freeform 
input received from a user, comprising the steps of: 

(a) accepting a freeform input, said input con- 
taining message information and a time refer- 

30 ence; 

(b) extracting said message information and 
said time reference from said freeform input; 

(c) providing an alert responsive to said time 
information. 

OS 

2. The method described in claim 1 , wherein said step 
of accepting an input includes one or more of scan- 
ning a document containing at least one freeform 
input, receiving said freeform input via a pen-based 

40 computer system, and receiving said freeform input 
via a touch-screen computer system. 

3. A method according to any of the preceding claims, 
wherein said step of extracting includes one or more 

45 of locating at least one annotation, preferably anno- 
tation lifting, transmitting a signal via a telephone, 
transmitting a signal via a pager, alerting a user via 
an audio signal, transmitting said message via a 
facsimile device, and transmitting said message via 

so a computer network. 

4. The method according to any of the preceding 
claims, further including the step of displaying said 
message on a video display. 

55 

5. A system for providing a reminder message based 
on a freeform input, said system comprising: 
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(a) a receiving means for receiving a freeform 
input containing message information and a 
time reference for delivery of said message; 

(b) a processing means coupled to said receiv- 
ing means for extracting said message informa- 5 
tion and said time reference from said freeform 
input; and 

(c) an alerting means for providing said mes- 
sage to the user in response to said time refer- 
ence. 10 

6. The 6y6tem as defined in claim 5, wherein said re- 
ceiving means includes one or more of a facsimile 
device, a digital scanner, means for reading input 
from a stylus, and a touch-screen computer system, 

7. A system according to claim 5 or claim 6, wherein 
said processing means includes a field finding 
means or an annotation lifting means. 

20 

8. A system according to any of claims 5 to 7, wherein 
said alerting means includes one or more of means 
for displaying said reminder message on a video 
display, means for generating an audio signal, 
means for delivering said reminder message via a 2S 
telephone, and means for delivering said reminder 
message via a pager. 



minder messag in respons to said time ref- 
erence, said second interface coupled to said 
processor. 

11. A pen-based computer system for providing a re- 
minder message based on a freeform digital ink in- 
put, said system comprising: 

(a) a first interface for receiving a freeform in- 
put, said freeform digital ink input containing 
message information and a time reference for 
delivery of said message; 

(b) a processor coupled to said first interface, 
said processor responsive to said first interface 
and programmed to extract said message in- 
formation and said time reference from said 
freeform digital ink input and to create a remind- 
er message based on said message informa- 
tion; and 

(c) a second interface for transmitting said re- 
minder message in response to said time ref- 
erence, said second interface coupled to said 
processor. 



A computer readable storage medium for use with 
computer apparatus, said medium including com- 30 
puter instructions which, when executed by said 
computer apparatus: 

(a) receive a freeform input from a user, said 
input including message information and a time & 
reference; 

(b) extract said message Information and said 
time reference from said freeform input; 

(c) construct a reminder message based on 
said message information; and 40 

(d) transmit said reminder ihkmessage respon- 
sive to said time reference. 



10. A system for providing a reminder message based 
on a freeform input, said system comprising: *s 

(a) a first interface for receiving a scanned im- 
age of a freeform input, said freeform input con- 
taining message information and a time refer- 
ence lor delivery of said message; so 

(b) a processor coupled to said first interface, 
said processor responsive to said first interface 
and programmed to extract said message in- 
formation and said time reference from 
scanned image of said freeform input and to 
create a reminder message based on said mes- 
sage information; and 

(c) a second interfac for transmitting said re- 
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